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ABSTRACT 



A parallel, scalable internetworking unit (lU) architecture 
employing at least two network controllers (NCs), a fore- 
ground buffer controller with local memory, a background 
buffer controller with local memory, a node processor (NP) 
and a buffer memory. Each network attached to the lU has 
an individual network controller which communicates with 
the foreground buffer controller. The foreground buffer 
controller interfaces with NCs and maintains queucing infor- 
mation. The background buffer controller communicates 
with the foreground buffer controller for maintairung pack- 
ets of data as linked lists of buffers in the buffer memory. The 
NP communicates with both the foreground and background 
buffer controllers to process stored header information. And, 
a connection matrix is provided to dynamically interconnect 
muldple lUs for increased parallel processing of packet 
trafQc and processing. 

12 Claims, 9 Drawing Sheets 
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APPARATUS HAVING THREE SEPARATED 
AND DECENTRALIZED PROCESSORS FOR 
CONCURRENTLY AND INDEPENDENTLY 
PROCESSING PACKETS IN A 
COMMUNICATION NETWORK 5 

FIELD OF THE INVENTION 

The invention relates generally to an intemeiworking unit 
architecture and, more particularly, to a scalable parallel iq 
internetworking unit architecture designed to parallel pro- 
cess the tasks of cpnunand/data trafficking, packet process- 
ing and bufifer management, 

BACKGROUND OF THE INVENTION 15 

The present invention involves the concept of networking 
networks, a procedure called network interconnection, or 
internetting. 

Internetting was first explored in the early 1970*s by 
DARPA. The agency sought ways to interconnect different 
packet networks so that computers using them could com- 
municate without concern for which kind of and how many 
networks made the interconnections. Special processors 
called gateways (or internetworking units) were developed 
to connea two or more networks and to pass packet traffic 
from one network to the next. 

To send the information, a processor creates packets 
including the source and destination addresses and encap- 
sulates the packets in the format required by its local 30 
network. The computer then routes the packets to the 
appropriate internetworking unit (I WU) for further process- 
ing and transmission. 

Two significant tasks of IWXJ' s have been identified as the 
buffering and switching of packets. The buffering task 35 
involves reception of packets into the main buffers and 
transmission of these packets out of the main buffers. The 
switching task is to determine the destination of the trans- 
mission and to modify the formats of the packets as required 
by the neighboring network protocols, 40 

The buffering and switching, commonly referred to as 
storc-and-forward, are necessary in the packet-switching 
environment due to network differences such as media 
speeds, operating protocols, and network traffic. 
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SUMMARY OF THE INVENTION 



One aspect of the present invention provides an internet- 
working unit OWU) for receiving and storing packets from 
a network, processing the packets and forwarding the pack- 50 
ets to another network. The IWU includes a packet memory 
for storing packets and a background buffer controller means 
coupled to the packet memory for organizing and maintain- 
ing the packets in the packet memory. Also included is a 
foreground buffer controller, where the background control- 55 
ler is coupled between the foreground controller and the 
packet memory, for transferring packets to and from said 
background controller. Each of the networks has a network 
controller and each netwoik controller is coupled to the 
foregroimd controller for controlling the transmission of go 
packets from a network to the foreground controller and the 
reception of packets from the foreground controller to a 
network. 

Another aspect of the present invention provides an 
internetworking uinil (IWU) for receiving and storing pack- 65 
ets from one network, processing the packets and forward- 
ing packets to another network. The IWU includes a plu- 



rality of packet memories for storing packets and a plurality 
of background buffer controllers, each coupled to an indi- 
vidual packet memory, for organizing and maintaining pack- 
ets. Also included is a plurality of foreground buffer con- 
trollers. A connection switch is coupled between the 
plurality of background controllers and the plurality of 
foreground controllers for transferring packets between any 
one of the foreground controllers and any one of the back- 
ground controllers. 

BRIEF DESCRIFnON OF THE DRAWINGS 

The invention is best understood from the following 
detailed description when read in connection with the 
accompanying drawings, in which: 

FIG. 1 shows a functional block diagram of a lowend 
scalable IWU system suitable for use with the present 
invention; 

FIG. 2 shows a high-level functional block diagram of a 
network controller (NC) suitable for use with the present 
invention; 

FIG. 3 shows a flowchart illustrating the processing steps 
of a node processor suitable for use in the lU of FIG. 1; 

FIG. 4 shows a functional block diagram of a highend lU 
system employing a plurality of the lowend lUs of FIG. 1; 

HG. 5 shows a format of a queue table entry (QTE) 
suitable for use in the systems of FIGS. 1 and 4; 

FIGS. 6(a) and 6(b) show the format of a packet table 
entry (PTE) suitable for use in the systems of FIGS. 1 and 
4; 

FIG. 7 shows the formal of a buffer table entry (BTE) 
suitable for use in the systems of FIGS. 1 and 4; 

FIG. 8 schematically represents the relationship between 
QTEs, PTEs and BTEs as shown in FIGS. 5, 6 and 7, 
respectively. 

FIG. 9 shows the details of the connection matrix of FIG. 

4; 

no. 10 shows an example of the timing of an operation 
of the highend lU of FIG. 4; 

FIG. 11 shows the throughput of the lU of FIG. 4 in Mega 
bits per second (Mbps) with 2 Fiber Distributed Data 
Interfaces (FDDIs); and 

FIG. 12 shows the throughput of the lU of FIG. 4 in Kilo 
packets per second (Kpps) analysis with 2 FDDIs. 

DETAILED DESCRIPTION OF THE INVENTION 
1. Overview 

The present invention provides an interaetworidng unit 
(lU) architecture which makes the storing and forwarding of 
packets efficient by . using various parallelisms among com- 
mand/data trafficking, packet processing and buffer manage- 
ment. An example of an internetworking unit architecture 
can be found in pending U.S. Pat. No. 5,367,643 which is 
herein incorporated by reference. 

By using the various parallelisms, the lU provides: 1) 
high-speed non-blocking data paths, 2) distributed buffer 
management control for concurrent packet receptions and/or 
transmissions and 3) separate storages and access paths for 
packet headers and payloads. • 

FIG. 1 shows a lowend scalable internetworking unit (lU) 
100 suitable for use with the present invention. 

lU 100 includes a plurality of specialized netwoik con- 
trollers (NCs) 102a, 102^ connected to an adaptor bus 
(GAB) 104. NCs 102a, 1026 provide a specialized intaface 
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bctw^n a network and lU 100 for sending and receiving 
packets of data. Each NC 102a, 102fe communicates with its ' 
respective network using that network's protocol and com- 
municates with other lU 100 processors using GAB 104 
protocol. 5 

Foreground buffer controller (FGAM) 106 is also con- 
nected to GAB 104, FGAM 106, operating as a frontend 
buffer control processor, is responsible for interfacing with 
the NCs using GAB 104 protocol and handling data read/ 
write operations. lo 

FGAM 106 also maintains, in queue memory (QM) 108, 
queues which indicate the status of received packets (i.e., 
indicate next packet for processing or next packet for 
transmitting). However, FGAM 106 has no knowledge 
regarding where or in what form the packets are stored in 15 
packet memory (PM) 112. 

It should be noted that creating a front end buffer con- 
troller which does not know where or in what form the 
packets arc stored effectively decentralizes the overall con- 
trol of the buffers. This decentralization of control allows for 20 
greater parallel processing of data transfer and control as 
well as greater scalability. 

Connected to FGAM 106 is a background buffer control 
(BGAM) 110 which maintains the storage of packets in the 
packet memory (PM) 112. BGAM 110 is responsible for 25 
splitting the packet into its respective parts, payload and 
header; therefore, packet memory 112 is split into two 
separate memories: PM/p 112ci is designated to store the 
payload information for received packets while PM/h 112fc 
is designated to store the header information for received 30 
packets. 

BGAM 110, operating as a backend processor, manages a 
free buffer list indicating which buffers in PM 112 are free 
as well as an in-use buffer list indicating which linked lists 
of buffers currently contain packets. The control information 35 
associated with the packets and buffers is maintained in 
packet buffer memory (PBM) 114. 

It should be noted, however, that BGAM 110 does not 
have direct access to GAB 104; thus, the packet movement 
involving BGAM 110, whether incoming or outgoing, 40 
passes through FGAM 106. Thus, the buffer management 
functions performed by BGAM 110 are isolated from GAB 
104. 

Connected, via RSC bus 120, to both FGAM 106 and 
BGAM 110 is a node processor (NP) 116. NP 116, using 45 
RSC bus 120, accesses and processes packet header infor- 
mation. 

lU 100 provides an architecture which allows data traffic 
between NCs 102, FGAM 106 and BGAM 110 to occur 
concurrently with header processing by NP 116 since NP 116 50 
can access packets in PM 112 without tying up GAB 104. 
Additionally, BGAM 110 can perform memory management 
functions (e.g., organizing and maintaining buffers) without 
interfering with GAB 104 activity. 

As mentioned above, lU 100 is lowend and scalable. 55 
Because the control of the buffers has been decentralized by 
creating an FGAM and a BGAM separate of one another, 
efficiently connecting multiple lowend lUs 100 is possible. 
These interconnections create a highend lU capable of 
handling more throughput than a lowend lU. 60 

To construct a highend lU, a plurality of lowend lUs 100 
are strategically interconnected with a connection matrix 
(CM) which is briefly described with reference to FIG. 4. 

CM 402 strategically and dynamically interconnects 
FGAMs 106a-n and NPs 116fl-rt with BGAMs UQa-n to 65 
produce a highend lU 400. This configuration (1) allows any 
FGAM 106a-n to store packets in any PM 112a~n via the 
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corresponding BGAM llOa-a (2) aUows any NP 116a-n lo 
access and process headers maintained by any BGAM 
llOfl-fi, and (3) allows any FGAM lQ6a~n to access the 
information maintained by any BGAM llOa-n for forward- 
ing purposes. 

n. Details of the Exemplary Embodiment 
A. Lowend Internetworking Unit (lU) 100 
The present invention provides an internetworking unit 
architecture which makes the storing and forwarding of 
packets cflBcicnt by exploiting various parallelisms among 
the command/data trafficking, the packet header processing 
and the buffer managing. 

1. Network Controllers (NCs) 102 

Referring back to FIG. 1, NCs 102a, 102b provide a 
specialized interface between a network and lU 100. 

Each NC 102a, 1026 is capable of communicating with its 
associated network using that network's protocol while also 
communicating with FGAM 106 using GAB 104 protocol. 
Some examples of networks for which a NC 102a, 1026 may 
interface include an FDDI, an Ethernet, a token ring and a 
microchannel. 

Additionally, each NC 102a, 1026 contains staging buff- 
ers (not shown) which are used to temporarily store data, 
either incoming or outgoing, in order to resolve potential 
mismatches between the network media speed and the 
internal bus speed of lU 100. The staging buffers are small 
in granularity so that a reception or transmission can be 
started as soon as possible. In this way, for either a reception 
or transmission, the latency delay introduced by a NC 102a 
or 1026 between the network port and PM 112 is minimized. 

Additionally, each NC 102a, 1026 is capable of issuing 
instructions to FGAM 106 regarding the starting of a recep- 
tion, allocation of unused buffers, enqueueing of received 
packets, retrieval of packets for transmission, releasing of 
buffers after successful packet transmissions and various 
other queueing status checking operations. 

In the exemplary embodiment of the present invention, 
NCs 102a, 1026 are implemented with an architecture as 
shown in FIG. 2. 

Two dual-ported memories 202, 204 provide the elasticity 
and staging buffers for the packet data flowing to and from 
an FGAM (shown in FIG. 2). Each of the data buffers has 
byte alignment logic 206, 208 at its input. The potential for 
specialized data filters 210, 212 exists on the output of either 
or both of the data buffers. 

The byte alignment logic 206 on the input of the outbound 
data buffer allows the transfer of packets which are aligned 
on arbitrary bytes in the PM (shown in FIG. 2). The byte 
alignment logic 208 on the input of the inbound data buffer 
allows for arbitrary alignment of bytes within words cither 
from the network interface, or from a local bus interface. 

The registers and counters, functional block 214, are used 
to generate addresses to the PM, and construct and field 
FGAM requests. 

2. Generic Adaptor Bus (GAB) 104 

Referring to FIG. 4. GAB 104a and GAB 1046 are the 
busses on which NCs 102a~n and FGAMs 106a-« commu- 
nicate. In the exemplary embodiment of the present inven- 
tion, lU 400 is capable of supporting multiple GABs 104a, 
1046 each having different speeds. GAB 104a is designed to 
operate with an 80 ns cycle while fast GAB 1046 is designed 
to operate at a 40 ns cycle. This capability is provided by 
way of FGAMs which can operate at either speed in order 
to be compatible with existing NCs as well as new NCs 
designed with new technologies. 

3. Foreground Buffer Control (FGAM) 106 
Referring to FIGS. 2 and 4, FGAM 106, as a frontend 

processor, is responsible for interfacing with NCs 102 using 
GAB 104 protocol and handling data read/write operations. 
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FGAM 106 also maintains the incoming and outgoing 
queues, and keeps the queueing information (or queue table 
entries [QTEs]) in the form of linked lists for the packets in 
queue memory (QM) 108. 

FIG. 5 shows the format of a QTE. A QTE comprises two 5 
sixteen bit words. For word 0. bits 15-14 indicate the packet 
type while bits 13-0 indicate the forward link. For word 1, 
bits 15-14 indicate trafBc type while bits 13-0 indicate the 
backward link. A QTE contains no packet retrieval infor- 
mation, thus, when an FGAM is communicating with a lo 
BGAM, it uses the QTE ID. 

4. Queue Memory (OM) 108 

Associated with each FGAM 106 is a queue memory 
(QM) 108. QM 108 is used to store the CJTEs as described 
above. 15 

5. Background Buffer Control (BGAM) 110 

BGAM 110 is connected to FGAM 106. BGAM UO, 
functioning as a backend processor, manages the free buffer 
list and the stored padcets as linked lists of buffers. 

The control information about packets (packet table 20 
entries [PTEs]) and buffers (buffer table entries [BTEs]) is 
maintained in PBM 114. 

FIGS. 6a and 6b show the format of a PTE in the free list 
and in use, respectively. FIG. 6a shows two 32-bit words. In 
word 0, bits 14-0 indicate the next free PTE pointer while the 25 
remaining bits of word 0 and word 1 are unused. 

FIG. 6b also shows two 32-bit words. In word 0, bits 
31-29 indicate the packet state, bits 28-24 are reserved, bit 
24 indicates if this PTE is for the first buffer in the group of 
buffers storing the packet, bits 23-16 indicate the number of 30 
buffen used to store the packet and bits 14-0 indicate the 
current BTE ID. In word 1, bits 30-16 indicate the ID of the 
first BTE for this packet and bits 14-0 indicate the ID of the 
last BTE for diis packet. 

FIG. 7 shows the format of a BTE. Each BTE is a 32-bit 35 
word wherein bits 31-24 indicate the buffer offset, bits 23-16 
indicate the byte count, bit 15 indicates the zero byte buffer 
and bits 14-0 indicate the next buffer pointer. 

FIG. 8 is a schematic representation of the relationships 
between the QTEs, the PTEs and the BTEs. As seen in HG. 40 
8, word 0 of CJTE 500a points to QTE SWib which points to 
QTE 500c, while word 1 of QTE 500c points to (JTE 500fc 
which points to QTE 500a. 

Each QTE maps directly to a PTE, For example, QTE 
500a maps directly to PTE 600a, and word 1 of PTE 600a 45 
contains the pointers to the first BTE TOOaj and the last BTE 
70003. 

The following example is described with reference to 
HG. 2 to illustrate the operation of the overall data slmcture 
and its relationships. When NC 102a wants to transmit a 50 
packet, it issues a Request-Reception (RRx) primitive to 
FGAM 106. FGAM 106 responds to NC 102a with a QTE 
ID which is a pseudo-buffer pointer. Although NC 102a 
interprets the QTE ID as the starting address of some buffer 
in a psuedo-PM, the data from NC 102a is written into 55 
FGAM 106. 

To obtain the QTE ID, FGAM 106 informs BGAM 110 of 
the start of a packet reception which, in turn, allocates a free 
PTE and an initial BTE. Since the BTE ID is directly 
mapped to the starting address of the buffer space, BGAM 60 
110 is ready to accept packet data. The PTE ID is then 
returned to FGAM 106 as the response from BGAM 110. 
FGAM 106 then uses the PTE ID to allocate the needed QTE 
(if in a highend IWU, a BGAM ID is also necessary). Once 
FGAM 106 receives data from NC 102a on GAB 104, 65 
FGAM gives the QTE ID to BGAM 110 in order to enable 
BGAM 110 to keep the proper context. 
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Every time a data movement is started, FGAM 106 also 
uses the offset provided by NC 102a to inform BGAM 110 
of the point at which to resume the data transfer. Note that 
BGAM 110 desirably keeps track of the current buffer (BTE) 
that the offset is related to. If more buffers are needed as the 
reception proceeds. NC 102a issues Request-Buffer (RB) 
primitives to get more buffers. But, FGAM 106 always 
responds with the same QTE ID as pseudo buffer pointers. 
That is, FGAM 106 does not keep track of the progress of 
the data transfer. BGAM 110 automatically allocates more 
free BTEs whenever it detects a zero offset passed from 
FGAM 106. 

Because all packet data movement is serviced by BGAM 
110 through a connection-oriented protocol, there is no need 
for an address bus between BGAM 110 and FGAM 106. 

6. Packet Memory (PM) 112 

Associated with each BGAM HO is a packet memory 
(PM) 112. PM 112 is split into two memories: PMyp 112a for 
storing the payload of a packet and PM/h 112^? for storing 
the header of a packet. 

PM/p 112a is organized as buffers of memory. In the 
exemplary embodiment, each buffer is 256 consecutive 
bytes. Likewise, PM/h 1126 is organized as buffers of 
memory, and each of its buffers is 32 consecutive bytes. 

7. Packet Buffer Memory (PBM) 114 

Also associated with each BGAM 110 is a packet buffer 
memory (PBM) 114. PBM 114 is used to store the PTEs and 
BTEs as described above. 

8. Node Processor (NP) 116 

NP 116 is connected to both FGAM 106 and BGAM 110 
via RSC bus. NP 116 processes packet headers which have 
been received and stored iri PM/h 112b, 

Referring back to Uie flowchart of FIG. 3, the processing 
steps basically include communicating with FGAM 106 to 
determine which packet is next to be processed, step 300. NP 
116 communicates, via RSC bus 120, with FGAM 106 in 
order to access its processor queue which contains the ID of 
. the next packet to be processed. 

Once NP 116 knows which packet is next, NP 116 
accesses the header for that packet through BGAM 110 
which controls buffer management, step 302. Using the 
packet ID, NP 116 makes a request, via RSC 120, to BGAM 
110 to retrieve the corresponding header fit)m PM/h 112fr. 

Next, NP 116 transforms the original header into a header 
compatible with the network for which the packet is des- 
tined, step 304. It should be noted tiiat the extent of NP 116 
conversion capabilities is application dependent As 
described below in the Applications section, whctiier an NP 
116 converts formats at the data link layer, the network layer 
or both depends on the type of networks that lU 400 
interconnects. 

NP 116 replaces the original header with die new header 
for that packet via BGAM 110, step 306. After processing, 
NP 116 then makes another request, via RSC bus 120, to 
BGAM 110 to replace the original header in PM/h I12b with 
the new converted header. 

Fmally. NP 116 makes a request, via RSC 120, to FGAM 
106 in order to enqueue that particular packet. In other 
words, NP 116 informs FGAM 106 that this particular 
packet is ready to be transnaitted, step 308. 

Because NP 116 performs translations from one. known 
network format to another known network format, an imple- 
mentation of NP 116 is weU known to those skilled in the art 
Detailed discussions of network architecture layers and 
associated protocols are found in G. E. Keiser, Local Area 
Networks, 1989 McGraw-Hill, and R. J. Cypscr, Communi- 
cations for Cooperating Systems, 1991 Addison- Wesley 
which are both herein incorporated by reference. 
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B. Highend lU 400 

Referring back to FIG. 4, to construct a highend lU, a 
plurality of lowend lUs 100 are strategically interconnected 
with a connection matrix (CM) 402. 

CM 402 strategically and dynamically interconnects s 
FGAMs 106a-n and NPs lUa-n with BGAMs llOa-n to 
produce a highend lU 400, Having the basic buffer controller 
responsibilides divided into an FGAM 106a arid a BGAM 
110a is particularly advantageous for a highend lU because 
it permits a configuration which, in parallel, (1) allows any 10 
FGAM l(^~n to store packets in any PM ll2a-n via the 
corresponding BGAM UOa-fi, (2) allows any NP I16a~-n to 
access and process the headers maintained by any BGAM 
110a-/i; and (3) allows any FGAM 106 to access the 
infonnation maintained by any BGAM llOa-n for forward- 15 
. ing purposes. 

1. Connection Matrix (CM) 402 

CM 402 and its operation are explained by way of an 
example. 

When a Request Reception conunand (RRx) is received 20 
by FGAM i, it then makes a connection request to CM 402 
for an arbitrary BGAM which is not assigned to any FGAM 
at that time. In this case, assume that CM 402 forwards the 
connection request and the RRx conunand to BGAM j. A 
connection is established when BGAM j grants the connec- 25 
tion request CM 402 holds the connection until FGAM i 
drops the request. Before storing the packet, BGAM j 
responds to the request of FGAM i with the number j along 
with the PTE ID so that the packet can be located later. 

CM 402 also accepts requests from FGAMs 106a-n to 30 
connect to a specific BGAM when doing transmission and 
header processing. When there are multiple FGAMs 106a-n 
wailing for the same BGAM llOo, a prioritized scheme can 
be implemented in which the more recent or less important 
user of a specific BGAM 110a has a lower priority. 35 

FIG. 9 shows a functional block diagram of CM 402. The 
matching policy is implemented by a centralized scheduler 
(SCHED) 902. Collecting the requests from FGAMs i06a-n 
(NPs 116a-n) and status information from BGAMs llOa-n, 
SCHED 902 generates signals to control isolators 904 40 
individually. There cannot be two FGAMs 106a-n or two 
NPs 116 accessing the same BGAM UOa-n. An FGAM 
106a-n should have no problem getting a buffer from a free 
BGAM llOa-n as long as there are as many BGAMs llOa-n 
as FGAMs \06a-n. 45 

2. Queueing and Service Bus (QSB) 404 

Referring back to FIG. 4. the above description of CM 
402 presents a scheme for how a receiving FGAM 106a-n 
stores packets into a PM 112a-7i. However, an FGAM 
106a-n can forward packet retrieval information to another 50 
FGAM 106 which is to handle a transmission by way of a 
queueing and service bus (QSB) 404. 

For example, once the destination of a packet is known by 
an FGAM 106a, with help from the associated NP 116a, 
FGAM 106a then uses QSB 104 to broadcast both the packet 55 
destination and retrieval infonnation. All FGAMs I06b-n 
are required to listen in while others are broadcasting. Only 
the FGAM 106b-n which is to handle the transmission takes 
the packet retrieval infonnation from the bus and attaches it 
to the respective outgoing queue. QSB 404 is also useful for 60 
other types of FGAM coordinations during the initialization, 
in. Operation 

A. Primitives 

The following is a list of primitives which define func- 
tionality and interactions among FGAMs 106a-n, BGAMs 65 
UOa-/i and CM 402. The primitives are explained in the 
context of actions performed by an FGAM 106 in conjunc- 
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tion with a connected BGAM 110. CM 402, once connec- 
tions are requested and granted via SCHED 902, is consid- 
ered transparent between FGAM 106 and BGAM 110. 

Accept: Each FGAM is assigned a set of queues at system 
initialization.- These queues are associated with NPs or 
PMIs. 

Create: An FGAM issues this primitive upon receiving 
requests from PMIs to receive or create packets. The FGAM 
sends a connection request to the SCHED and sends the 
primitive Uirough CM. The connected BGAM responds with 
its ID and a free packet ID number (PID). The FGAM 
records the (BGAM ID, packet ID) pair that uniquely 
identifies a QTE in QM. The BGAM allocates a PTE from 
the head of the free PTE pool. The allocated PTE is set to be 
a single buffer packet The BGAM then appends this PTE ID 
after its own ID to form the unique QTE ID and responses 
to the FGAM. The BGAM also prc-allocates a free BTE 
from the free BTE pool. 

Write: The FGAM issues this primitive when transferring 
packet data into the PM. It is done by first requesting, for 
connection, the BGAM specified by the QTE ID of the 
packet. Once connected, the PID is sent to the BGAM 
followed by the packet data. Following the initial primitive, 
packet data is streamed to the BGAM one word per cycle. 
After receiving this primitive, the BGAM uses the PID to 
retrieve the PTE and writes the data into the PM. A counter 
is used to accumulate the number of bytes of data received. 
When the current buffer is filled up to some threshold, a new 
buffer is allocated from the free BTE pool. Once a new 
buffer is used, the following operations are performed on the 
PTE: 

(1) update die last BTE ID of the PTE to reflect packet 
growth; 

(2) increment the number of buffers of PTE by one; 

(3) record the byte count and offset of the previous 
finished buffer, and 

(4) increase the packet length by the value of the counter 
register and reset the counter to zero. 

The write primitive can be issued any number of times 
during packet generation depending on the scheduling of the 
FGAM-BGAM connections. The current pointer is updated 
only when the data streaming stops. Also, the packet length 
is updated by the counter value. Note that the counter is not 
used if the packet is not in the packet generation stage. 

Open: The FGAM issues this primitive to service the 
packet at the head of the queue. According to the QTE ID of 
die packet, the FGAM asks for connection to the specified 
BGAM, and passes the request to "open a packet" to the 
BGAM. When the BGAM receives this primitive, it 
retrieves the PTE of the packet. It then resets the current 
pointer to the beginning of the packet to prepare for the 
ensuing read primitives. PS is also modified if necessary to 
reflect the packet* s new status. 

Read: The FGAM issues this primitive when reading 
packet data. The remaining packet length of the specified 
packet is expected from the BGAM before any data. Upon 
encoimtering an end-of-packet indicator, the FGAM stops 
reading and drops the connection request. After receiving 
this primitive, the BGAM uses the received PID to retrieve 
the PTE and obtain the current pointer for data being read 
from the PM. The BGAM first calculates the remaining 
packet length according to the first BTE ID, last BTE ID and 
cunent pointer. The remaining length is returned to the 
FGAM as the first response.- The BGAM then begins to 
supply data words one cycle at a time. Before the current 
buffer is exhausted, the next buffer is located by tracing 
down die BTE chain. The cunent pointer points to the next 
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data to be read. Similar to the write primitive, the read 
primitive is usually issued more than once during a packet 
consumption period. 

Purge: Due to some exception conditions, such as a PMI 
being overrun by the incoming data from the medium, 5 
packets under generation or processing may be dropped. A 
connection request is issued according to the QTE ID to the 
intended BGAM. Upon receiving this primitive, the BGAM 
frees the specified PTE to the free PTE pool and returns all 
the allocated BTE's to the free BTE pool. lo 

Close: When a FGAM is done with a packet, the FOAM 
queues the packet to a NP or NC. If the specified queue is 
local to the FGAM, it links the QTE into the double linked 
list of the local queue. Otherwise, the FGAM requests the 
ownership of the QSB-Bus. Once granted, the FGAM broad- 15 
casts the destination queue ID followed by the QTE ID on 
the bus. All FGAMs are constantly monitoring the QSB- 
Bus. If the broadcast queue is found local, the QTE ID is 
copied and linked in the specified queue. No BGAMs are 
involved. 20 

Erase: This primitive has two types, delete from the 
packet head or tail Packet data is deleted from the head if 
the packet header has shrunken, and frx)m the tail if data is 
to be removed, for example, a frame check sequence (PCS). 
It should be noted that the FCS is usually stripped off by the 25 
NC. According to the received PID, the packet length is 
decreased by the given number of bytes, BTE is updated to 
reflect the result of this reduction. The first BTE ID or the 
last BTE ID are changed if this reduction causes an empty 
buffer. 30 

Lseek: An FGAM issues this primitive to set the current 
pointer of the PTE in the BGAM to an offset relative to the 
current current pointer position. It should be noted that both 
positive and negative offsets are allowed. According to the 
QTE ID, FGAM requests a connection to the BGAM 35 
overseeing the packet. Once granted, offset value and type 
are sent to BGAM. BGAM updates the current pointer value 
of the packet by adding the offset to current pointer. If the 
resulting cunenl pointer is negative, this primitive adds 
extra buffers in die beginning of the BTE chain. Note that 40 
this primitive only increases the packet space. Valid data can 
be filled by issuing the write primitives. 

B. Example 

HG. 10 shows the steps, in machine cycles, of a reception 
of a 16-byte packet from a NC in order to illustrate, by 45 
example, the interactions among the functional components 
of highend lU 400. 

First, the NC starts with a GAB bus request (BR) signal 
and waits for the bus grant (BG) signal from the FGAM. 
Once the BG is detected, the NC holds the bus ownership in 50 
the next cycle by activating the bus busy (BSY) signal, and 
gives out a RRx signal. After the detection of the RRx, the 
FGAM issues a connection request (CR) signal to the CM 
asking for a free BGAM, and forwards the RRx. The 
connection establishment is indicated to the FGAM by a 55 
coimecdon grant (CG) signal. At the same time, the RRx 
signal is forwarded to the BGAM. The response (res), which 
consists of the BGAM ID and a packet ID, is then propa- 
gated back to the FGAM and the NC in two cycles. After the 
response is generated, the BGAM begins some house keep- 60 
ing work, such as preparing the next PTE and BTE. and 
recording the first two words of the PTE information. 

When the response returns to the NC, the NC writes the 
first 4-byte word (dl) to the FGAM, The FGAM then issues 
a CR signal and a write request (Wrq) signals to die CM. The 65 
Wrq signal is then forwarded to the BGAM at the same time 
the CG signal is generated from the CM to the FGAM. After 
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the FGAM detects the CG signal, a response (rl) is sent back 
to the PMI, and the data word is forwarded to the BGAM. 
The data words and responses between the FGAM and the 
BGAM are denoted by D#*s and R#*s, respectively. 

Finally, the NC issues an enqueue (ENQ) signal, which 
will be propagated to the BGAM^ causing the last piece of 
the PTE information to be recorded. The FGAM in the 
propagation path activates the QSB bus request (QBR) 
signal after the confirming CG signal from the CM. After the 
FGAM gets the QSB grant (QBG) signal, it then issues a 
queueing control (QBC) signal, which contains the address 
of the packet and the address of the transmission queue. All 
FGAM's listen to the QBC signal, and the one which 
manages the transmission queue takes the information from 
the bus to enqueue the packet in QTE format The operation 
is complete by preparing a free QTE and the FGAM is ready 
for the next enqueue operation. 

C. Applications 

Referring to HG. 4, lU 400 can be used differentiy in 
different applications. 

For example, lU 400 can be used as a router. If lU 400 is 
employed as a network router between two fiber distributed 
data interfaces (FDDIs) with a common network layer 
protocol, because the formats do not differ, an NP 116 need 
only modify the destination address in the packet header 
before the packet is forwarded. 

Another example is lU 400 used as a gateway. An lU 400 
can be employed as a gateway between an FDDI network 
using TCP/IP and an Ethernet network using XNS/IDP. In 
this case, the formats differ up to and including the network 
layer. In Operation, an Ethernet frame arrives at its associ- 
ated NC, the Ethernet frame encapsulation format is stripped 
off by die NC and the data packet is stored in an IDP format. 
Next, an NP 116 converts the format of the packet from IDP 
to TCP/IP with appropriate modifications to the destination 
address. Finally, the TCP/IP packet is forwarded to its 
destination, where the intermediate NC encapsulates it in an 
FDDI format. 

In the above examples, the NCs are designed to strip off 
encapsulating formats or add encapsulating formats associ- 
ated wiUi the data link layer for data coming into an lU or 
out of an lU, respectively. However, these stripping and 
adding fiinctions can be implemented in an NP such that the 
NP docs a format conversion for each of die various layers 
(e.g., data link, network, transport). 

IV. Performance 

Tests with the present architecture have been conducted 
using Fiber Distributed Data Interface (FDDI) networks as 
the target attachments. 

Various combinations of average packet interarrival times . 
and packet data lengths of exponential distribution are used 
to generate network traffic. The throughput is measured by 
aggregate packet receptions from all NCs to determine the 
performance of the architecture, 

FIGS. 11 and 12 show the aggregate throughput in units 
of Mbps (megabits per second) and Kpps (thousand packets 
per second) for an lU. The results indicate that the lU is 
capable of handling all FDDI traffic types without being a 
bonleneck. 

Although the invention is illustrated and described herein 
embodied as a parallel scalable internetworking unit archi- 
tecture, the invention is nevertheless not intended to be 
limited to the details as shown. Rather, various modifications 
may be made in the details within the scope and range of 
equivalents of the claims and without departing from the 
spirit of the invention. 

What is claimed: 

1. In a system having a plurality of networks communi- 
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eating with packets, a parallel, scalable inter-networking 
packet routing apparatus for receiving and storing packets 
from one of the plurality of networks, processing the packets 
and forwarding packets to" another one of the plurality of 
networks comprising: 5 
a plurality of first control means each first control means 

for receiving packets from anyone of the plurality of 

networks, for maintaining a queuing status of the 

received packets and transferring packets; 
a plurality of packet memoiy means for storing packets 

received and transferred by the plurality of first control 

means; 

a plurality of second control means, separate from the 
plurality of first control means, each second control 
means coupled to an individual one of said plurality of 
packet memory means, for receiving the transferred 
packets from a first control means, and for organizing 
and maintaining packets in respective packet memory 
means, the plurality of second control means operate 
concurrently with and independcndy from the plurality 
of first control means; 

connection switch means, coupled between the plurality 
of first control means and the plurality of second 
control means, for transfering packets between any one 25 
of the plurality of first control means and any one of the 
plurality of second control means; and 

a plurality of node processor means coupled to a respec- 
tive first control means and said connection switch 
means, each node processor means for 1) obtaining 30 
status of the stored packets from the respective first 
control means, 2) accessing packets from a packet 
memory means by way of the connection switch means 
and a second control means, 3) processing packets 
based on their origin and destination, 4) storing the 35 
processed packets back into the packet memory means 
by way of the connection switch means and the second 
control means and 5) for updating the queueing status 
of the packets by way of the respective first control 
means, the plurality of node processors operate con- 40 
currently with and independently from the first and 
second control means. 

2. The inter-networking packet routing apparatus of claim 
1 further providing each of said plurality of networks having 

a network controller means, each network controller means 45 
coupled to one of said plurality of first control means, each 
of said network controller means for controlling the trans- 
mission of packets from its respective networic to said first 
control means and the reception of packets from said first 
control means to its respective network. 50 

3. The inter-networking packet routing apparatus of claim 
1 further providing a plurality of queue memories, each 
coupled to one of said plurality of first cotrol means and 
containing a queuing status in the form of queue table entries 
(QTEs) indicating which packets are ready to be processed 55 
by a respective node processor means and which packets are 
ready to be transferred from a respective packet memory 
means to one of said plurality of network controller means. 

4. The inter-networking packet routing apparatus of claim 

3 wherein providing a plurality of packet buffer memories 60 
which are used in the maintenance and organization of said 
packets, each coupled to one of said plurality of second 
control means and containing packet table entries (PTEs) 
and buffer table entries (BTEs), each PTE points to a linked 
list of BTEs which correspond to a plurality of locations in 65 
a respective packet memory means containing the packet 
identified by a QTE. 
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5. The inter-networking packet routing apparatus of claim 
1 wherein said packets comprise a pay load and a header, 
each of said packet memory means comprises first and 
second memories, wherein said first memory contains 
packet payloads and said second memory contains packet 
headers. 

6. The inter-networking packet routing apparatus of claim 
1, wherein packets are received from one of said plurality of 
networks and packets are sent to another of said plurality of 
networks, wherein all of said first control means are coupled 
via a communication bus. 

7. In a system having a plurality of nctworics communi- 
cating with packets, a parallel, scalable inter- networking 
packet routing apparatus for receiving and storing packets 
from one of the plurality of networks, processing the packets 
and forwarding the packets to another of the plurality of 
networks comprising: 

first control means for receiving packets from one of said 
plurality of networks, for maintaining a queuing status 
of the received packets and transferring the packets, 

packet memory means for storing packets received and 
transferred by the first control means; 

second control means, separate from the first control 
means and coupled between the first control means and 
the packet memory means, for receiving the transferred 
packets from the first control means, and for organizing 
and maintaining the packets in the packet memory 
means, the second control means operates concurrently 
with and independently from the first control means; 

node processor means, separate from the first and second 
control means and coupled to both the first and the 
second control means for 1) obtaining status of the 
stored packets from the first control means, 2) access- 
ing packets from the packet memory means by request 
to the second control means, 3) processing packets 
based on their origin and destination, 4) storing the 
processed packets back into the packet memory means 
by way of the second conu-ol means and 5) for updating 
the queueing status of the packets by way of the first • 
control means, the node processor operates concur- 
rently with and independently from the first and second 
control means, 

thereby the first control means, second control means and 
the node processor means being separated and decen- 
tralized of control to provide for parallel operation and 
scalabiUty. 

8. The inter-networking packet routing apparatus of claim 
7, wherein said node processor means is direcdy coupled to 
said second control means and said first control means. 

9. The inter-networking packet routing apparatus of claim 
7 further providing each of said plurality of networks having 
network controller means, each network controller means 
coupled to said first control means for controlling the 
transmission of packets from one of said plurality of net- 
works to said first control means and the reception of packets 
from said first control means to one of said plurality of 
networks. 

10. The inter-networking packet routing apparatus of 
claim 7 further providing a queue memory coupled to said 
first control means, said queue memory containing queuing 
status in the form of queue table enuies (QTEs) indicating 
which packets are ready to be processed by said node 
processor means and which packets are ready to be U^s- 
ferred from said packet memory means to one of said 
plurality of network controller means. 

11. The inter-networking packet routing apparatus of 
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claim 10 further providing a packet buffer memory coupled 
to said second control means which is used in the mainte- 
nance and organization of said packets, said packet buffer 
memory containing packet table entries (PTEs) and buffer . 
table entries (BTEs), each PTE points to a linked list of 
BTEs which correspond to a plurality of locations in the 
packet memory means containing the packet identified by a 
QTE. 
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12. The inter-networking packet routing apparatus of 
claim 7, wherein said packets comprise a payload and a 
header, said packet memory means comprises first and 
second memories, said first memory contains packet pay- 
loads and said second memory contains packet headers. 
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